Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Latest commit

 

History

History
History

README.md

Outline

PySpark Benchmarks

This directory contains microbenchmarks for PySpark using ASV (Airspeed Velocity).

Prerequisites

Install ASV:

pip install asv

For running benchmarks with isolated environments (without --python=same), you need an environment manager. The default configuration uses virtualenv, but ASV also supports conda, mamba, uv, and some others. See the official docs for details.

Running Benchmarks

All commands below can be run from the Spark root directory using ./python/asv, which is a wrapper that forwards arguments to asv in the benchmarks directory.

Quick run (current environment)

Run benchmarks using your current Python environment (fastest for development):

./python/asv run --python=same --quick

You can also specify the test class to run:

./python/asv run --python=same --quick -b 'bench_arrow.LongArrowToPandasBenchmark'

Full run against a commit

Run benchmarks in an isolated virtualenv (builds pyspark from source):

./python/asv run master^!          # Run on latest master commit
./python/asv run v3.5.0^!          # Run on a specific tag
./python/asv run abc123^!          # Run on a specific commit

Compare two commits

Compare current branch against upstream/master with 10% threshold:

./python/asv continuous -f 1.1 upstream/master HEAD

Other useful commands

./python/asv check          # Validate benchmark syntax

Writing Benchmarks

Benchmarks are Python classes with methods prefixed by:

  • time_* - Measure execution time
  • peakmem_* - Measure peak memory usage
  • mem_* - Measure memory usage of returned object

Example:

class MyBenchmark:
    params = [[1000, 10000], ["option1", "option2"]]
    param_names = ["n_rows", "option"]

    def setup(self, n_rows, option):
        # Called before each benchmark method
        self.data = create_test_data(n_rows, option)

    def time_my_operation(self, n_rows, option):
        # Benchmark timing
        process(self.data)

    def peakmem_my_operation(self, n_rows, option):
        # Benchmark peak memory
        process(self.data)

See ASV documentation for more details.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.