MDPSolver (mdpsolver) is a Python package for large Markov Decision Processes (MDPs) with infinite horizons. This wiki includes details about the documentation of MDPSolver and other information, such as performance tests.

• Fast solver: Our C++-based solver is substantially faster than other MDP packages available for Python, depending on the problem size and parameters. See the Performance tests section in this wiki.
• Available on PyPI.
• Two optimality criteria: Discounted and Average reward.
• Three optimization algorithms: Value iteration, Policy iteration, and Modified policy iteration.
• Three value-update methods: Standard, Gauss–Seidel, and Successive over-relaxation.
• Uses span norm or supremum norm stopping criterion depending on the selected update method.
• Supports sparse matrices and parallel computing.
• Operating systems: Linux and Windows.

Install directly from PyPI with:

pip install mdpsolver

Some users will encounter the version 'GLIBC_2.32' not found error when attempting to import MDPSolver in Python. The issue can be resolved by manually compiling and replacing the SO-file for the optimization module in the MDPSolver package. Follow these steps to compile, install, and test the module.

Requires Visual Studio 2022 (17.9) with MSVC C++ compiler and libraries installed (see below).

After installing Visual Studio (incl. MSVC C++ compiler and libraries), install directly from PyPI with:

pip install mdpsolver

• Operating system: Windows or Linux.
• Windows users must install Visual Studio 2022 incl. the MSVC C++ compiler (see the installation instructions above).
• Linux users must ensure that GLIBC 2.38 or newer is installed (check your version with ldd --version).

Andersen et al., (2025). MDPSolver: An Efficient Solver for Markov Decision Processes. Journal of Open Source Software, 10(109), 7544, https://doi.org/10.21105/joss.07544

BibTeX

@article{Andersen2025,
        doi = {10.21105/joss.07544},
        url = {https://doi.org/10.21105/joss.07544},
        year = {2025}, publisher = {The Open Journal},
        volume = {10},
        number = {109},
        pages = {7544},
        author = {Anders Reenberg Andersen and Jesper Fink Andersen},
        title = {MDPSolver: An Efficient Solver for Markov Decision Processes},
        journal = {Journal of Open Source Software} }