If you use SolverBenchmark.jl in your work, please cite using the format given in CITATION.cff.
This package provides general tools for benchmarking solvers, focusing on the following guidelines:
- The output of a solver's run on a suite of problems is a
DataFrame, where each row is a different problem.- Since naming issues may arise (e.g., same problem with different number of variables), there must be an ID column;
- The collection of two or more solver runs (
DataFrames), is aDict{Symbol,DataFrame}, where each key is a solver;
This package is developed focusing on Krylov.jl and JSOSolvers.jl, but is sufficiently general to be used in other places.
Obs: See the assets folder for the complete code, or the docs for a more detailed example.
pretty_stats(df)
┌─────────────┬─────────┬───────────┬───────────┬────────┐
│ flag │ name │ f(x) │ time │ iter │
├─────────────┼─────────┼───────────┼───────────┼────────┤
│ failure │ prob001 │ -6.89e-01 │ 6.24e+01 │ 70 │
│ failure │ prob002 │ -7.63e-01 │ 3.53e+02 │ 10 │
│ first_order │ prob003 │ 3.97e-01 │ 7.68e+02 │ 10 │
│ first_order │ prob004 │ 8.12e-01 │ 4.31e+01 │ 80 │
│ first_order │ prob005 │ -3.46e-01 │ 2.68e+02 │ 30 │
│ first_order │ prob006 │ -1.88e-01 │ 6.68e+01 │ 80 │
│ first_order │ prob007 │ -1.61e+00 │ 1.57e+02 │ 60 │
│ first_order │ prob008 │ -2.48e+00 │ 6.05e+02 │ 40 │
│ first_order │ prob009 │ 2.28e+00 │ 1.36e+02 │ 40 │
│ failure │ prob010 │ 2.20e-01 │ 8.38e+02 │ 50 │
└─────────────┴─────────┴───────────┴───────────┴────────┘
pretty_latex_stats(df)
df = join(stats, [:status, :f, :t], ...)
pretty_stats(df, tf=markdown)
| id | name | flag_alpha | f_alpha | t_alpha | flag_beta | f_beta | t_beta | flag_gamma | f_gamma | t_gamma |
|--------|---------|-------------|-----------|-----------|-------------|-----------|-----------|-------------|-----------|-----------|
| 1 | prob001 | failure | -6.89e-01 | 6.24e+01 | first_order | -4.83e-01 | 3.92e+02 | failure | -9.99e-01 | 6.97e+02 |
| 2 | prob002 | failure | -7.63e-01 | 3.53e+02 | first_order | -1.16e+00 | 4.79e+02 | first_order | 1.03e+00 | 4.35e+02 |
| 3 | prob003 | first_order | 3.97e-01 | 7.68e+02 | first_order | -2.14e-01 | 6.82e+01 | first_order | -1.16e+00 | 9.86e+02 |
| 4 | prob004 | first_order | 8.12e-01 | 4.31e+01 | first_order | -1.37e+00 | 4.80e+02 | first_order | 5.34e-01 | 9.97e+02 |
| 5 | prob005 | first_order | -3.46e-01 | 2.68e+02 | first_order | -1.54e+00 | 4.68e+02 | first_order | -3.08e-01 | 5.08e+02 |
| 6 | prob006 | first_order | -1.88e-01 | 6.68e+01 | first_order | -1.23e+00 | 4.52e+02 | first_order | 9.86e-01 | 2.16e+02 |
| 7 | prob007 | first_order | -1.61e+00 | 1.57e+02 | first_order | -1.96e+00 | 6.44e+02 | first_order | -1.19e+00 | 8.59e+02 |
| 8 | prob008 | first_order | -2.48e+00 | 6.05e+02 | failure | -4.73e-01 | 6.69e+02 | first_order | 6.80e-01 | 9.05e+02 |
| 9 | prob009 | first_order | 2.28e+00 | 1.36e+02 | first_order | 1.34e+00 | 9.48e+01 | failure | 2.04e-03 | 4.35e+02 |
| 10 | prob010 | failure | 2.20e-01 | 8.38e+02 | first_order | 8.08e-01 | 9.49e+02 | first_order | -4.78e-01 | 6.59e+01 |
pretty_latex_stats(df)
p = performance_profile(stats, df->df.t)
p = profile_solvers(stats, costs, titles)
If you think you found a bug, feel free to open an issue. Focused suggestions and requests can also be opened as issues. Before opening a pull request, start an issue or a discussion on the topic, please.
If you want to ask a question not suited for a bug report, feel free to start a discussion here. This forum is for general discussion about this repository and the JuliaSmoothOptimizers organization, so questions about any of our packages are welcome.