QuantumForest is a new lib on the model of differentiable decision trees. It has the advantages of both trees and neural networks. Experiments on large datasets show that QuantumForest has higher accuracy than both deep networks and best GBDT libs(XGBoost, Catboost, mGBDT,...).

• Keeping simple tree structure，easy to use and explain the decision process

• Full differentiability like neural networks. So we could train it with many powerful optimization algorithms (SGD, Adam, …), just like the training of deep CNN.

• Support batch training to reduce the memory usage greatly.

• Support the end-to-end learning mode. No more work to preprocess data.

  

To verify our model and algorithm, we test its performance on five large datasets.

The following table listed the accuracy of some libraries

*Some results are copied form the testing results of NODE

All libraries use default parameters. It’s clear that mGBDT needs much more memory than other libs. mGBDT always failed because out or memory for most large datasets. Both NODE and QuantumForest have higher accuracy than CatBoost and XGBoost. It is a clear sign that differentiable forest model has more potential than classical GBDT models.

python main_tabular_data.py

Please use the following bibtex entry:

[1] Chen, Yingshi. "Deep differentiable forest with sparse attention for the tabular data." arXiv preprint arXiv:2003.00223 (2020).
[2] Chen, Yingshi. "LiteMORT: A memory efficient gradient boosting tree system on adaptive compact distributions." arXiv preprint arXiv:2001.09419 (2020).

• More huge testing datasets.

  ​ If anyone has larger dataset, please send to us for testing

• More models.

• More papers.

The provided implementation is strictly for academic purposes only. If anyone is interested in using our technology for any commercial use, please contact us.

Yingshi Chen (gsp.cys@gmail.com)