Java开源机器学习框架Datumbox发布0.6.0版本

主要的新内容?

新版0.6.0 的主要焦点是扩展框架来处理大型数据、 改进的代码体系结构和公共的 Api、 简化数据解析、 加强了文档内容和移动到宽松的许可证。
以下是详细内容:

1. Handle Large Data: The improved memory management and the new persistence storage engines enabled the framework to handle big datasets of several GB in size. Adding support of the MapDB database engine enables the framework to avoid storing all the data in memory and thus be able to handle large data. The default InMemory engine is redesigned to be more efficient while the MongoDB engine was removed due to performance issues.
2. Improved and simplified Framework architecture: The level of abstraction is significantly reduced and several core components are redesigned. In particular the persistence storage mechanisms are rewritten and several unnecessary features and data structures are removed.
3. New “Scikit-Learn-like” public APIs: All the public methods of the algorithms are changed to resemble Python’s Scikit-Learn APIs (the fit/predict/transform paradigm). The new public methods are more flexible, easier and more friendly to use.
4. Simplify data parsing: The new framework comes with a set of convenience methods which allow the fast parsing of CSV or Text files and their conversion to Dataset objects.
5. Improved Documentation: All the public/protected classes and methods of the Framework are documented using Javadoc comments. Additionally the new version provides improved JUnit tests which are great examples of how to use every algorithm of the framework.
6. New Apache License: The software license of the framework changed from “GNU General Public License v3.0” to “Apache License, Version 2.0“. The new license is permissive and it allows redistribution within commercial software.

Since a large part of the framework was rewritten to make it more efficient and easier to use, the version 0.6.0 is not backwards compatible with earlier versions of the framework. Finally the framework moved from Alpha into Beta development phase and it should be considered more stable.

How to use it

In a previous blog post, we have provided a detailed installation guide on how to install the Framework. This guide is still valid for the new version. Additionally in this new version you can find several Code Examples on how to use the models and the algorithms of the Framework.

Next steps & roadmap

The development of the framework will continue and the following enhancements should be made before the release of version 1.0:

1. Using Framework from console: Even though the main target of the framework is to assist the development of Machine Learning applications, it should be made easier to be used from non-Java developers. Following a similar approach as Mahout, the framework should provide access to the algorithms using console commands. The interface should be simple, easy to use and the different algorithms should easily be combined.
2. Support Multi-threading: The framework currently uses threads only for clean-up processes and asynchronous writing into disk. Nevertheless some of the algorithms can be parallelized and this will significantly reduce the execution times. The solution in these cases should be elegant and should modify as little as possible the internal logic/maths of the machine learning algorithms.
3. Reduce the use of 2d arrays & matrices: A small number of algorithms still uses 2d arrays and matrices. This causes all the data to be loaded into memory which limits the size of dataset that can be used. Some algorithms (such as PCA) should be reimplemented to avoid the use of matrices while for others (such as GaussianDPMM, MultinomialDPMM etc) we should use sparse matrices.

Other important tasks that should be done in the upcoming versions:

1. Include new Machine Learning algorithms: The framework can be extended to support several great algorithms such as Mixture of Gaussians, Gaussian Processes, k-NN, Decision Trees, Factor Analysis, SVD, PLSI, Artificial Neural Networks etc.
2. Improve Documentation, Test coverage & Code examples: Create a better documentation, improve JUnit tests, enhance code comments, provide better examples on how to use the algorithms etc.
3. Improve Architecture & Optimize code: Further simplification and improvements on the architecture of the framework, rationalize abstraction, improve the design, optimize speed and memory consumption etc.

As you can see it’s a long road and I could use some help. If you are up for the challenge drop me a line or send your pull request on github.