Open
Description
Sometimes it is convenient to group parametrized tests by function name, to look at the total time taken by the group of parametrized tests, rather than individual parametrized tests. When trying multiple combinations of parameters, it is easy to generate a big number of parametrized tests without realizing the total test time.
cc @jeremiedbb, something like this could be added CI if we think this is worth it. Right now we are only showing the 20 slowest individual tests we could add the 20 slowests tests with grouped parametrized tests.
To write the tests timings info to a file, add this to sklearn/conftests.py:
def pytest_terminal_summary(terminalreporter, exitstatus, config):
# use @ as CSV delimiter , ; and | are arising in test names so can not be used ...
lines = ['name@duration\n']
for reps in terminalreporter.stats.values():
for rep in reps:
if getattr(rep, 'when', None) == "call":
line = f"{rep.nodeid}@{rep.duration}\n"
lines.append(line)
with open('/tmp/test_timings.csv', 'w') as test_timings_file:
test_timings_file.writelines(lines)To analyse the CSV:
import pandas as pd
pd.options.display.max_colwidth = None
df = pd.read_csv('/tmp/test_timings.csv', delimiter='@')
df['function_name'] = df['name'].str.extract(r'(.+?)\[.+\]')
df['function_name'] = df['function_name'].where(df['function_name'].notna(), df['name'], axis=0)
print(df.groupby('function_name')['duration'].aggregate(['sum', 'size']).sort_values(by='sum', ascending=False).iloc[:50])On my machine:
sum size
function_name
sklearn/tests/test_common.py::test_estimators 90.395519 7291
sklearn/cluster/tests/test_k_means.py::test_k_means_fit_predict 25.514541 32
sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch 23.514811 108
sklearn/tests/test_common.py::test_search_cv 11.408637 640
sklearn/tree/tests/test_tree.py::test_prune_tree_regression_are_subtrees 10.413731 88
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_kind 9.233481 24
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_partial_dependence_plot_limits_one_way 7.611308 12
sklearn/inspection/tests/test_partial_dependence.py::test_output_shape 7.524255 144
sklearn/tests/test_common.py::test_pandas_column_name_consistency 7.299764 229
sklearn/decomposition/tests/test_dict_learning.py::test_minibatch_dictionary_learning_positivity 6.513909 12
sklearn/linear_model/tests/test_quantile.py::test_quantile_estimates_calibration 6.451284 3
sklearn/model_selection/tests/test_split.py::test_stratified_kfold_ratios 6.122864 28
sklearn/tests/test_docstrings.py::test_docstring 6.090240 1803
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_lines_kw 5.780402 12
sklearn/tree/tests/test_tree.py::test_prune_tree_classifier_are_subtrees 5.337500 36
sklearn/decomposition/_dict_learning.py::sklearn.decomposition._dict_learning.DictionaryLearning 5.231721 1
sklearn/utils/tests/test_estimator_checks.py::test_check_estimator_clones 4.757799 1
sklearn/tests/test_common.py::test_check_n_features_in_after_fitting 4.365753 195
sklearn/tests/test_calibration.py::test_calibration_multiclass 4.298814 8
sklearn/ensemble/tests/test_forest.py::test_memory_layout 4.225799 8
sklearn/_loss/tests/test_loss.py::test_loss_dtype 3.981264 1024
sklearn/neighbors/tests/test_neighbors.py::test_knn_forcing_backend 3.858353 20
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_partial_dependence_overwrite_labels 3.802329 12
sklearn/cluster/tests/test_k_means.py::test_predict 3.793582 24
sklearn/tests/test_docstring_parameters.py::test_fit_docstring_attributes 3.763307 203
sklearn/inspection/tests/test_partial_dependence.py::test_partial_dependence_dataframe 3.611114 12
sklearn/linear_model/tests/test_quantile.py::test_sparse_input 3.534900 18
sklearn/mixture/tests/test_gaussian_mixture.py::test_init 3.497789 1
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_passing_numpy_axes 3.449946 6
sklearn/ensemble/tests/test_forest.py::test_sparse_input 3.325918 15
sklearn/linear_model/tests/test_sgd.py::test_SGDClassifier_fit_for_all_backends 3.193899 3
sklearn/metrics/tests/test_pairwise_distances_reduction.py::test_strategies_consistency 3.174157 56
sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_sample_weight_effect 3.005303 6
sklearn/ensemble/tests/test_forest.py::test_importances 2.923119 20
sklearn/linear_model/tests/test_ridge.py::test_ridge_gcv_vs_ridge_loo_cv 2.905850 48
sklearn/cluster/tests/test_optics.py::test_dbscan_optics_parity 2.799847 9
sklearn/inspection/_plot/tests/test_plot_partial_dependence.py::test_plot_partial_dependence_str_features 2.790852 26
sklearn/manifold/tests/test_spectral_embedding.py::test_spectral_embedding_precomputed_affinity 2.748455 8
sklearn/model_selection/tests/test_successive_halving.py::test_aggressive_elimination 2.747456 8
sklearn/neural_network/tests/test_mlp.py::test_n_iter_no_change_inf 2.743698 1
sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_missing_values_resilience 2.473407 6
sklearn/impute/tests/test_impute.py::test_iterative_imputer_no_missing 2.403574 1
sklearn/ensemble/tests/test_weight_boosting.py::test_adaboost_consistent_predict 2.401502 2
sklearn/linear_model/tests/test_ridge.py::test_ridge_gcv_sample_weights 2.219063 32
sklearn/decomposition/tests/test_dict_learning.py::test_warning_default_transform_alpha 2.208973 2
sklearn/linear_model/tests/test_quantile.py::test_equivariance 2.128632 3
sklearn/neighbors/tests/test_neighbors.py::test_neigh_predictions_algorithm_agnosticity 1.918901 168
sklearn/linear_model/tests/test_quantile.py::test_quantile_sample_weight 1.890437 1
sklearn/metrics/tests/test_pairwise_distances_reduction.py::test_pairwise_distances_radius_neighbors 1.859356 56
sklearn/experimental/tests/test_enable_iterative_imputer.py::test_imports_strategies 1.771305 1